Assisting input from a keyboard

ABSTRACT

Assisting input from a keyboard is described. In an embodiment, a processor receives a plurality of key-presses from the keyboard comprising alphanumeric data for input to application software executed at the processor. The processor analyzes the plurality of key-presses to detect at least one predefined typing pattern, and, in response, controls a display device to display a representation of at least a portion of the keyboard in association with a user interface of the application software. In another embodiment, a computer device has a keyboard and at least one sensor arranged to monitor at least a subset of keys on the keyboard, and detect an object within a predefined distance of a selected key prior to activation of the selected key. The processor then controls the display device to display a representation of a portion of the keyboard comprising the selected key.

BACKGROUND

Typing using a keyboard has remained as the most common way of inputtingalphanumeric data into a computer. The keyboard can be characterized asan indirect interaction device, as the user interacts with a separatedevice (the keyboard) and the result of the interaction (the text input)is displayed on-screen spatially offset from the keyboard. Even whensoftware keyboards are used with touch-sensitive screens, the input isstill indirect to some extent as the fingers are pressing keys that areoffset from a text input location (i.e. cursor) on the screen. If asoftware keyboard is displayed sufficiently close to the text inputlocation to be considered as direct interaction, then the keyboardconsequently obscures a significant portion of the user interface.

The indirect nature of keyboard input means that users can oftenstruggle to learn to enter alphanumeric data in an efficient manner.This is often because the users are not sufficiently familiar with thekeyboard layout, which results in the users looking at the keyboardrather than the text being entered on the screen. This is exacerbated bythe proliferation of new types of devices having keyboards, in bothhardware and software forms. This includes, for example, tablets,smartphones and netbooks. Each of type of device has its own designconstraints, which results in keyboard layout differences.

Often, as a result of unfamiliarity with the keyboard layout, users makerepeated mistakes when entering text using a keyboard, and as a result amuscle memory develops. Such a muscle memory can be hard to subsequentlybreak.

The embodiments described below are not limited to implementations whichsolve any or all of the disadvantages of known keyboard data entrytechniques.

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding to the reader. This summary is not anextensive overview of the disclosure and it does not identifykey/critical elements of the invention or delineate the scope of theinvention. Its sole purpose is to present some concepts disclosed hereinin a simplified form as a prelude to the more detailed description thatis presented later.

Assisting input from a keyboard is described. In an embodiment, aprocessor receives a plurality of key-presses from the keyboardcomprising alphanumeric data for input to application software executedat the processor. The processor analyzes the plurality of key-presses todetect at least one predefined typing pattern, and, in response,controls a display device to display a representation of at least aportion of the keyboard in association with a user interface of theapplication software. In another embodiment, a computer device has akeyboard and at least one sensor arranged to monitor at least a subsetof keys on the keyboard, and detect an object within a predefineddistance of a selected key prior to activation of the selected key. Theprocessor then controls the display device to display a representationof a portion of the keyboard comprising the selected key.

Many of the attendant features will be more readily appreciated as thesame becomes better understood by reference to the following detaileddescription considered in connection with the accompanying drawings.

DESCRIPTION OF THE DRAWINGS

The present description will be better understood from the followingdetailed description read in light of the accompanying drawings,wherein:

FIG. 1 illustrates a computing device with an on-screen keyboardrepresentation;

FIG. 2 illustrates a flowchart of a process for displaying a keyboardrepresentation;

FIG. 3 illustrates a capacitive key sensor;

FIG. 4 illustrates an infra-red key sensor;

FIG. 5 illustrates a laser key sensor;

FIG. 6 illustrates highlighting of a key in the representation prior toa key-press;

FIG. 7 illustrates highlighting of a key without a full representationprior to a key-press;

FIG. 8 illustrates highlighting of a key with a portion of therepresentation prior to a key-press;

FIG. 9 illustrates an exemplary computing-based device in whichembodiments of the keyboard assist technique can be implemented.

Like reference numerals are used to designate like parts in theaccompanying drawings.

DETAILED DESCRIPTION

The detailed description provided below in connection with the appendeddrawings is intended as a description of the present examples and is notintended to represent the only forms in which the present example may beconstructed or utilized. The description sets forth the functions of theexample and the sequence of steps for constructing and operating theexample. However, the same or equivalent functions and sequences may beaccomplished by different examples.

Although the present examples are described and illustrated herein asbeing implemented in a desktop computing system, the system described isprovided as an example and not a limitation. As those skilled in the artwill appreciate, the present examples are suitable for application in avariety of different types of systems using keyboard input.

Reference is first made to FIG. 1, which illustrates a computing devicecomprising a keyboard 100, a processing unit 102, and a display device104. Although the computing device shown in FIG. 1 is shown havingseparate elements (as in a desktop computing device), the elements canalso be integrated into a single unit (as in a laptop/notebook computingdevice, tablet or smartphone). The keyboard shown in FIG. 1 has atraditional QWERTY layout, but in other examples any type of keyboardlayout can be used. Examples of alternative layouts include AZERTY,QWERTZ or Dvorak. In addition, the keyboard can be of a device-specificlayout, for example a keyboard mounted on the rear of a tablet devicearranged to be used whilst holding the device.

The processing unit 102 comprises a processor and can optionally furthercomprise storage devices and interfaces, as outlined in more detail withreference to FIG. 9 below. The processing unit 102 is arranged toreceive input from the keyboard, execute instructions (in the form ofsoftware) and provide output to the display device 104. In this example,the processing unit 102 is executing an operating system (OS) andapplication software. In this example, the application software is aword-processing application, but any application using keyboard inputcan also be used.

The display device 104 can be in the form of a screen arranged todisplay a graphical user interface to a user of the computing device. Inthis example, the display device 104 is displaying a user interface 106of the application software executed on the processing unit 102.

In use, a user of the computing device can enter alphanumeric data intothe keyboard 100 by typing with first hand 108 and second hand 110. Thekeyboard 100 provides key-activation data corresponding to keys pressedby the user to the processing unit 102. The processing unit 102processes the key-activation data and, in the case that the datacorresponds to alphanumeric information intended to be entered into theapplication software, controls the display device 104 to display the keypresses as characters 112 in the user interface 106.

If the user does not know the layout of the keyboard 100, then, in orderto type accurately, the user looks at the keyboard 100 and their hands108, 110, rather than at the user interface 106. This interrupts thetyping process, and slows it down. To address this, the processing unit102 is arranged to control the display device 104 to display arepresentation 114 (or visualization) of the keyboard layout. Therepresentation 114 is overlaid on the user interface 106, and positionedin proximity to the characters 112. The user is then able to view thekeyboard layout without looking at the keyboard 100. This enables theuser to become familiar with the keyboard layout, thereby increasing theuser's typing speed and accuracy.

However, the overlaying of the representation 114 obscures a portion ofthe user interface 106. This can result in the user being unable to seeadditional text or controls without either scrolling through the text,or moving the representation 114 in the display. This can be alleviatedby the processing unit rendering the representation 114 as partiallytransparent and/or by automatically rearranging the elements of the userinterface 106 and the representation 114 so that the representation 114only obscures areas of the user interface 106 which are not the currentfocus of the user's attention or input.

More experienced users who are familiar with the keyboard layout may notwant the representation 114 to be present, and FIG. 2 illustrates aflowchart of a process for automatically determining whether to displaythe representation 114, based upon the typing behavior of the user.

The flowchart of FIG. 2 illustrates a process performed by theprocessing unit 102. The processing unit 102 executes 200 theapplication software (e.g. word processing software as shown in FIG. 1),and receives 202 key-press input comprising alphanumeric data intendedfor the application software. The key-press input results from the usertyping using the keyboard 100 (i.e. the user activating a plurality ofkeys). The processing unit 102 then analyzes 204 the key-press input toidentify predefined typing patterns. The processing unit 102 can searchfor one or more known typing patterns in the key-press input, examplesof which are outlined in more detail hereinafter.

If it is determined 206 that one or more of the predefined typingpatterns have been detected, then the processing unit 102 controls thedisplay device 104 to display 208 the representation 114 of the keyboardto the user, overlaid on the application software user interface 106. Inother examples, the processing unit 102 can additionally control thedisplay of the representation 114 in order to reduce the amount of userinterface 106 obscured by the representation 114, as describedhereinafter.

If it is determined 206 that one or more of the predefined typingpatterns have not been detected, then the processing unit 102 returns toreceiving key-press input without displaying the representation 114, andthe process is repeated until a typing pattern is detected. Note thatwhilst the process shown in FIG. 2 is executed, the key-press input fromthe keyboard is also passed to the application if appropriate, so thatthe characters 112 are shown in the user interface 106 and areunaffected by the FIG. 2 process. Note that control keys can also bepart of the key-press input, but are not directly shown in theapplication but instead have other effects on the user interface orapplication software.

As mentioned above, the processing unit 102 analyzes 204 the key-pressinput to identify predefined typing patterns. Some example typingpatterns that the processing unit 102 can detect are now outlined. Inone example, the processing unit 102 can detect whether the key-pressescomprise a plurality of key-presses of a predefined key within a timethreshold. In other words, if the predefined key is pressed more thanonce within a certain time interval, then the display of therepresentation 114 is triggered. This can be used to detect, forexample, repeated pressing of the backspace key (or delete key) within atime interval. Repeated pressing of the backspace key in a shorttime-frame can indicate that the user is making many mistakes whentyping.

Pressing and holding a predefined key can be interpreted in a similarmanner, as the key-repeat function of the keyboard treats this in asimilar way to repeated key presses. In some examples, the time intervalcan be sufficiently short that only successive presses of the predefinedkey (without intermediate key-presses) triggers the display of therepresentation.

In another example, the processing unit 102 can detect whether thekey-presses comprise a key-press of a first predefined key immediatelyfollowing one or more key-presses of further predefined keys. In otherwords, certain combinations of key-presses can trigger the display ofthe representation 114. For example, the processing unit 102 can store alist of common typing mistakes made by users (e.g. typing ‘tje’ insteadof ‘the’), compare the list to the key-presses and detect when thekey-presses match an entry in the list. In addition, this type of typingpattern can also be provided by the application software itself (e.g. asprovided by Microsoft® Word's ‘autocorrect’ feature). In such examples,an interface to the application software is provided to enable theapplication software (rather than dedicated typing analysis algorithms)to indicate that the user would benefit from visualization assistancewith their typing at that time.

Alternatively (or additionally) the processing unit can track the user'styping behavior and learn the key combinations that indicate errorsbeing made. For example, the processing unit 102 can record when acertain combination of letters are typed, then deleted and replaced witha different combination (or the same letters in a different order). Thedeleted combination can be stored and used to trigger the display of therepresentation 114.

In another example, the processing unit 102 can determine from thekey-presses a rate of input of key-presses and detect whether this inputrate is less than a threshold value. Therefore, the processing unit 102uses the key-presses to estimate a typing speed. If the typing speed istoo low, then this can indicate that the user is struggling to type, andhence the display of the representation is triggered.

In another example, the processing unit 102 can determine whether thekey-presses comprise one or more words that are not present in a storeddictionary. In other words, the key-presses are compared to adictionary, such as that used by a spell-checker, to determine whethertyping mistakes are being made. Spell-checkers are widely used byword-processing and office software, and contain comprehensivedictionaries. These can therefore be used as an reliable determinationof whether the user's typing is accurate.

In another example, the predefined typing pattern can be an indicator ofthe user's typing style, and the triggering of the display of therepresentation 114 can be based on an estimation of this typing style.For example, an analysis can be made as to whether the user istouch-typing, or whether the user is a ‘hunt and peck’ typist (orsomewhere between these extremes). To achieve this, the processing unitcan record and analyze time-delays between key-presses and use thisinformation with knowledge of the keyboard layout to detect whether auser of the keyboard is not touch-typing. If the user is nottouch-typing, then the representation 114 can be displayed.

The relative timing between key-presses can be used to estimate how manyfingers the user is using to type. For example, for a touch-typist usingall fingers there is only a short time delay between pressing the ‘f’key and pressing the ‘a’ key (as an example only) as these keys areactivated by different fingers. Conversely, for a single-finger typist,these keys are pressed by the same finger, and hence there is arelatively long time delay between the key-presses. The processing unit102 can track one or more such key combinations and the time delaysbetween them to ascertain the typing style of the user, and display therepresentation 114 accordingly.

Alternatively (or in addition to) using the time-delays between certainkey combinations, the processing unit 102 can also determine whether theuser is touch-typing by using one or more sensors to detect the positionof the user's digits (i.e. fingers and thumbs) over the keyboard.Example sensors are discussed in more detail with reference to FIGS. 3to 5 below. By using sensors to track the position of the user's digits,the processing unit 102 can determine whether the user rests theirdigits on the ‘home keys’, and types from that position (indicatingtouch typing), or whether the user tends to move only a subset of theirdigits over the keys to type. As above, the representation 114 can bedisplayed to non-touch-typists to assist them with learning a moreefficient typing style.

Furthermore, this technique can be used to improve the typing abilitiesof users that can touch-type, but do not do so in the most efficient orcorrect manner. For example, the processing unit 102 can use the sensorsto detect that user is using the wrong finger to press a given key, andconsequently use the representation to indicate the correct finger tothe user. Therefore, even if the user is typing reasonably fast andaccurately, this improves the user's technique even further byencouraging the user to learn to touch-type correctly.

Note that any of the above-described techniques for detecting a typingpattern and triggering the display of the representation 114 can be usedeither alone or in combination.

As mentioned above with reference to FIG. 2, the processing unit 102 isarranged to display 208 the representation 114 on the display device104. The display of the representation 114 can be controlled in severalways, as outlined below, to enhance the usability of the computingdevice.

Firstly, in one example, the degree of transparency of therepresentation 114 can be dynamically controlled. For example, thedegree of transparency of the representation 114 can be dependent on theelapsed time since the most recent typing pattern (as described above)was detected (i.e. dependent on the frequency of typing errors). Inother words, when the predefined typing pattern is detected, therepresentation 114 is displayed with a first level of transparency, andas time elapses without further predefined typing patterns beingdetected the transparency of the representation 114 is increased untilit has completely disappeared from the display. If further predefinedtyping patterns are detected, however, the transparency of therepresentation 114 can be reduced to make it more visible to the user.

For example, the transparency of the representation can be controlledsuch that after one of the predefined typing patterns are detected thedegree of transparency of the representation 114 is increased by a fixedamount (e.g. after every detected typing mistake the representation 114becomes 5% less transparent). Alternatively, the transparency of therepresentation can be controlled such that after one of the predefinedtyping patterns are detected the representation 114 is reset to a fixeddegree of transparency (e.g. after every detected typing mistake therepresentation 114 changes to 50% transparent).

The transparency of the representation can be arranged to increaselinearly over time whilst the predefined typing patterns are notdetected. Alternatively, the transparency of the representation can bearranged to increase in discrete steps each time a specified intervalhas elapsed.

Dynamically controlling the transparency of the representation 114reduces the extent to which the representation obscures the userinterface 106 for user's that do not benefit from typing assistance(ultimately removing it altogether from the display), whilst ensuringthat it is made more prominent for user's that do require assistance. Inaddition, the changing transparency provides the user with encouragementin the form of a goal (making the representation completelytransparent), thereby encouraging them to avoid making mistakes andhence developing good typing habits.

In addition to controlling the transparency of the representation 114,the processing unit 102 can also (or alternatively) control the displayof the representation such that only a portion of the keyboard isdisplayed. For example, the amount of the keyboard shown in therepresentation can be made dependent on the elapsed time betweendetecting the predefined typing patterns (i.e. the frequency of typingerrors). If the elapsed time between detecting the predefined typingpatterns is less than a threshold, then the whole of the keyboard can berepresented, as this indicates that the user would benefit fromassistance. If the elapsed time is greater than this threshold, thenonly a portion of the keyboard can be displayed. By only displaying arepresentation of a portion of the keyboard, the user interface 106 isobscured less for users that do not require as much typing assistance.

The representation of a portion of the keyboard displayed can becentered on the area of the keyboard at which a typing error occurred.For example, when the processing unit 102 detects that the backspace keyis being used, the representation can be controlled to only comprisethose keys being deleted, and the immediately surrounding keys (or afixed size region surrounding these keys). Furthermore, if theprocessing unit 102 knows of alternative ‘correct’ typing patterns, e.g.through the use of a dictionary, then it can additionally highlight keysin the representation that the user intended to type.

In addition, the amount of the keyboard shown in the representation canbe controlled in dependence on the elapsed time between detecting thepredefined typing patterns. For example, if only infrequent errors aremade, then only a small area of the keyboard is represented (e.g. onlythe key pressed in error and its surrounding keys). If more frequenterrors are made, then successively larger portions of the keyboard canbe displayed in the representation (e.g. two levels of surrounding keys,or the half of the keyboard in which the error was made).

In another example, the overall size of the representation 114 can bedynamically controlled. As with the control of the transparency, thesize of the representation 114 can be controlled in dependence on theelapsed time since the most recent typing pattern was detected (i.e.dependent on the frequency of typing errors). In other words, when thepredefined typing pattern is detected, the representation 114 isdisplayed with a first size, and as time elapses without furtherpredefined typing patterns being detected the size of the representation114 reduces until it has completely disappeared from the display. Iffurther predefined typing patterns are detected, however, the size ofthe representation 114 can be increased to make it more visible to theuser.

The representation can be arranged to shrink linearly over time whilstthe predefined typing patterns are not detected. Each time thepredefined typing pattern is detected the representation can eitherrevert to a fixed size or increase in size by set amount.

In other examples, the location at which the representation is displayedon the display device can be dynamically controlled in dependence on thefrequency of detection of the predefined typing patterns. For example,if the predefined typing patterns are detected frequently (i.e. the useris making lots of typing errors) then the representation can be movedcloser to the text entry location (i.e. the cursor). Conversely, if thepredefined typing patterns are detected less frequently (i.e. the useris making fewer typing errors) then the representation can be movedfurther away from the text entry location, so that the user interface isless obscured.

In a further example, the information displayed within therepresentation can be controlled in dependence on the detected typingpatterns. For example, the key-press input from the user can bemonitored to determine whether the user is utilizing shortcut keys tocontrol the application software (e.g. ‘ctrl’+‘c’ to copy; ‘ctrl’+‘a’ toselect all, etc). If it is detected that shortcuts are not being used,then the representation can be changed to display the function of theshortcut when an appropriate key is pressed. For example, when the‘ctrl’ key is pressed, the representation can be changed to show theword ‘copy’ on the ‘c’ key, etc.

Furthermore, the shortcut keys can be actively promoted to the usersusing the representation. For example, if the processing unit 102detects that the user is using a pointing device to perform an action(such as copy, paste etc), then it can display the representation andhighlight the appropriate key combination to achieve the same action(e.g. ‘ctrl’+‘c’; ‘ctrl’+‘v’, etc). In this way, the users are educatedon the available shortcut keys for the actions that they perform.

Note that any of the above-described techniques for controlling thedisplay of the representation 114 can be used either alone or incombination.

In some examples, the process of FIG. 2 and the above-describedtechniques for triggering and displaying the representation can beperformed by a background process or software daemon, which can beincorporated into the operating system or can be a separately-installedapplication for the purpose of assisting typing and executed on theprocessing unit 102. By performing this process at the operating system,it enables the representation to be displayed for any applicationsoftware executed at the processing unit 102. In other examples, theapplication software can integrate the analysis techniques andvisualizations, so that users using that application software are aidedwithout relying on the presence of separately installed software. Asmentioned above, these techniques can be applied for any type ofkeyboard layout on any suitable device. This includes both hardwarekeyboards using physical switch devices, and also software keyboardswhere the keyboard is displayed on a touch sensitive display screen.

Reference is now made to FIGS. 3 to 8, which illustrate the use of keysensors to further enhance the use of the representation in assistingkeyboard input. In these examples, the sensors are arranged to detectwhen a digit of the user is within a predetermined distance (i.e.greater than or equal to zero) of a key. In other words, the sensors arearranged to detect either when a digit of the user is in close proximityto (but not yet in contact with) a key (i.e. ‘hovering’), or touching akey of the keyboard but not yet pressing the key sufficiently toactivate it (i.e. not applying sufficient pressure for key activation).

The sensors are arranged to provide the processing unit 102 withinformation on where the user's digits are over (or on) the keyboard andutilize this in the representation before a key is pressed. This enablesthe representation to be enhanced to show what the user is about to do,in advance of a key-press. This can therefore prevent the user fromperforming an erroneous action on the keyboard, which avoids amuscle-memory developing for an incorrect movement.

In one example, the sensors can be arranged to monitor all the the keyson the keyboard, to provide a full picture of how the user isinteracting with the keyboard. In alternative examples, the sensors cancover only a subset of the keys. For example, to determine whether ornot a user is touch-typing, only a subset of the keys can be monitored(e.g. the home keys). Alternatively, sensors can be used that are notprecise enough to determine which exact key each digit of the user ison, but are sufficiently precise to determine, for example, whether theuser is touch typing or not.

FIGS. 3 to 5 show example sensor arrangements. FIG. 3 illustrates acapacitive key sensing arrangement. A key-cap 300 is mounted on a switchdevice 302 in accordance with a traditional keyboard. Mounted on the topsurface of the key-cap 300 is a capacitive sensor 304. When a digit 306of the user is brought into contact with the capacitive sensor 304, achange in capacitance is detected, and the presence of the user's digit306 in contact with the key can be ascertained. The detection of theuser's digit 306 using a capacitive sensor 304 can be performed withoutthe user applying significant pressure to the key-cap 300, and hence thedigit 306 can be detected without activating the key. In addition, thecapacitive sensor 304 can also be used to detect when the digit 306 isin proximity to, but not touching, the key by increasing theamplification of the signals from the capacitive sensor. In analternative arrangement to that shown in FIG. 3, the capacitive sensorcan be located below the key rather than on the key cap, if asufficiently sensitive sensor is used. By applying the sensor shown inFIG. 3 to at least a subset of the keys of a keyboard, the user's digitscan be tracked, and the information used as described below.

FIG. 4 illustrates an infra-red (IR) key sensing arrangement. In a firstexample, the key-cap 300 comprises one or more (IR) emitters 400 and oneor more IR detectors 402. The IR detectors 402 are connected to acircuit (or software module) arranged to trigger a signal when theincident IR light on the detector is greater than a threshold. As theuser's digit 306 is brought closer to the key-cap 300, an increasingamount of IR light is reflected from the emitters 400 by the digit 306.When the digit is sufficiently close to the key-cap 300 the IR lightincident on the detectors 402 is sufficient to trigger the signal, andindicate the presence of the user's digit above the key. The thresholdcan be altered such that the user's digit is only detected when it isjust in contact with the key-cap 300, or when it is hovering above thekey-cap 300.

In another example, the emitters 400 from within the key-cap 300 canomitted, and emitters 404 mounted between the keys can be used instead.This can reduce the number of emitters used and the complexity of thekey-cap. In a further example, the emitters can be omitted completely,and passive IR detection can be used from the radiated heat of theuser's digit. In an alternative arrangement to that shown in FIG. 4, thekey caps can be transparent to IR light, and the sensors located beneaththe keys. By applying the sensors shown in FIG. 4 to at least a subsetof the keys of a keyboard, the user's digits can be tracked, and theinformation used as described below.

FIG. 5 illustrates a laser key sensing arrangement. A laser 500 isarranged to generate a sheet of light over the surface of the keyboard(i.e. above the key-cap 300). A lens can be used to generate such asheet from a laser emitter. An IR laser can be used, as this is notvisible to the human eye. A camera (e.g. an IR camera) is mountedrelative to the laser 500 so that reflected radiation can be detected bythe camera. For example, as shown in FIG. 5, the camera is mounted closeto the laser 500, so that when the sheet is interrupted by the digit 306of the user, the reflected radiation is visible to the camera 502.

Image data from the camera is passed to image processing software (whichcan be executed on the processing unit 102, or on a dedicatedprocessor), which uses the intensity of the reflected radiation toestimate the distance between the camera and the digit causing thereflection. This information can be combined with information on thekeyboard layout to determine which key the user's digit 306 is above.

The sensing arrangement of FIG. 5 does not require a separate sensor foreach key, but instead uses a single sensor to cover the whole keyboard.Furthermore, the sensing arrangement of FIG. 5 can be readily applied tosoftware keyboards, as no modification to the internal structure of thekeyboard is used.

In an alternative arrangement, the camera 502 of FIG. 5 can be utilizedwithout the laser 500. For example, the camera 502 can be arranged tooperate in a ‘shadow’ mode in which there is background light (eitherambient or provided by an emitter) and the digits of the user obscurethis light, such the image at the camera is a negative. This image canbe analyzed using image processing software to estimate the location ofthe digits of the user. In a further alternative, the laser 500 can bereplaced with less directional emitter, so that more of the digits areilluminated. The camera 502 then operates in ‘reflective’ mode, and theimage of the illuminated digits is analyzed using image processingsoftware to estimate their location. The ‘shadow’ and ‘reflective’ modeexamples can utilize either IR or visible light cameras. In a furtheralternative, the camera 502 can be a depth-camera. In the aboveexamples, the camera 502 can be arranged to view the keys from acrosstheir surface, or from the top. In addition, the camera can eitherintegrally mounted on the keyboard, or separate.

A further sensing arrangement (not shown in the figures) can utilizeadditional switches for each key, which detect the application of lightpressure on the key-cap, such as is seen when the user rests a digit ona key, but is not applying enough pressure to activate the key. Thisenables detection of when a user is lightly touching a key.

FIGS. 6 to 8 illustrate examples of how the key-sensing data can be usedto enhance the representation of the keyboard. As mentioned, the sensorsprovide data to the processing unit 102 that enable the processing unit102 to determine which keys are being touched or have a digit inproximity (depending on the sensing type). FIG. 6 illustrates a firstexample of how this data can be used to enhance the representation 114by highlighting a key in the representation prior to a key-press. In theexample of FIG. 6, the user is moving digit 600 such that it is hoveringabove (or touching) the ‘T’ key on the keyboard 100. However, the userhas not yet activated the ‘T’ key. The processing unit 102 has used thesensor data to detect the digit 600 in proximity to the ‘T’ key, and iscontrolling the display of the representation 114 to visually highlightthe ‘T’ key 602 in the representation. Therefore, the user is able tosee which key they are about to press in the representation 114, beforeactually activating the key. This enables the user to stop a movementbefore making a mistake, and may prevent an incorrect muscle memory fromdeveloping.

The processing unit 102 can be arranged to highlight all keys in therepresentation 114 that the sensors have determined have digits inproximity. Alternatively, as shown in FIG. 6, only a key that hasrecently detected a digit can be highlighted. In other words, theprocessing unit 102 attempts to highlight only those keys to which auser is currently moving their digits. Once a digit has been stationaryon a key for a predetermined period of time, the processing unit 102removes the highlighting for that key from the representation. Thisprevents the user being distracted by highlighted keys that are notcurrently being used, or are being touch by other parts of the user'shand.

FIG. 7 illustrates a further example of the use of key sensor data. Inthis example, the sensor data is used to control the display such thatonly the keys that have digits in proximity to them are displayed. InFIG. 7, the user's digit 600 is again touching (or hovering over) the‘T’ key, and the display device 104 displays a representation of the ‘T’key 700, but not a representation of the wider keyboard. This thereforestill provides the user with information on the key they are about topress, and prevents mistakes, but obscures very little of the userinterface 106. As with FIG. 6, the processing unit 102 can show all keysthat have digits in proximity, or only those for which movement isdetected (i.e. recent touching/hovering).

FIG. 8 illustrates a further example between the two extremes of FIGS. 6and 7. In this example, the sensor data is used to control the displaysuch that the keys that have digits in proximity to them are displayed,along with a region surrounding those keys. In FIG. 8, the user's digit600 is again touching (or hovering over) the ‘T’ key, and the displaydevice 104 displays a partial representation 800 of the keyboard showinga predefined radius centered on the ‘T’ key. In this example, the ‘T’key within the partial representation 800 is highlighted. This providesthe user with information on the key they are about to press, andprevents mistakes, and obscures less of the user interface 106 than afull representation of the keyboard. In addition, it still aids a userthat is unfamiliar with the keyboard layout, as they are able to see thesurrounding keys.

The users are also able to ‘search’ for a key by drawing a digit overthe keyboard, such that the partial representation reveals the keys inthe region under the user's digit, in the manner of a flashlightrevealing a portion of a darkened room. As above, the processing unit102 can show all keys that have digits in proximity, or only those forwhich movement is detected (i.e. recent touching/hovering).

The processing unit 102 can further utilize the sensor data by trackingthe detected digits and generating a model of the user's hands on thekeyboard. From the sensor data and the keyboard layout, the processingunit 102 can estimate which digits are in proximity to which keys, anduse this to render a representation of the user's hands. Therepresentation of the user's hands can be displayed on therepresentation 114 of the keyboard, to further help the user invisualizing how they are typing whilst looking at the display. Inaddition, the processing unit 102 can also generate a second set ofhands showing a preferred typing position for each key pressed by theuser (i.e. showing where the hands ought to be placed and what digitought to be used to activate a given key). This can be overlaid on therepresentation 114 of the keyboard in addition to the representation ofthe user's hands, so the user can compare their own hand positions tothe preferred hand positions.

FIG. 9 illustrates various components of an exemplary computing-baseddevice, such as processing unit 102, which can be implemented as anyform of a computing and/or electronic device, and in which embodimentsof the keyboard typing assistance technique can be implemented.

The processing unit 102 comprises one or more processors 902 which canbe microprocessors, controllers or any other suitable type of processorsfor processing computing executable instructions to control theoperation of the device in order to act upon inputs from a keyboardand/or sensors and display the representation accordingly.

The processing unit 102 comprises one or more input interfaces 904 whichare of any suitable type for receiving data from user input devices suchas keyboard 100, key sensors and/or a pointing device. An outputinterface 906 is arranged to output display information to displaydevice 104 which can be separate from or integral to the processing unit102. The display information provides the graphical user interface 106.Optionally, a communication interface 908 can be provided for datacommunication with one or more networks, such as the internet.

Platform software comprising operating system 912 or any other suitableplatform software can be provided at the computing-based device toenable application software 914 to be executed on the processing unit102. Other software functions can comprise one or more of:

-   -   A representation display module 916 arranged to control the        display device 104 to display the keyboard representation        overlaid on the user interface (with varying degrees of        transparency, extent, size and location);    -   A representation triggering module 918 arranged to determine        whether to display the representation based on the user typing        behavior;    -   A sensor mapping module 920 arranged to determine whether a        digit of the user is in proximity to a key on the keyboard;    -   A data store 921 arranged to store keyboard layouts, typing        patterns, dictionaries, etc.

The computer executable instructions can be provided using anycomputer-readable media, such as memory 910. The memory is of anysuitable type such as random access memory (RAM), a disk storage deviceof any type such as a magnetic or optical storage device, a hard diskdrive, or a CD, DVD or other disc drive. Flash memory, EPROM or EEPROMcan also be used. Although the memory is shown within the processingunit 102 it will be appreciated that the storage can be distributed orlocated remotely and accessed via a network or other communication link(e.g. using communication interface 908).

The term ‘computer’ is used herein to refer to any device withprocessing capability such that it can execute instructions. Thoseskilled in the art will realize that such processing capabilities areincorporated into many different devices and therefore the term‘computer’ includes PCs, servers, mobile telephones, personal digitalassistants and many other devices.

The methods described herein may be performed by software in machinereadable form on a tangible storage medium. Examples of tangible (ornon-transitory) storage media include disks, thumb drives, memory etcand do not include propagated signals. The software can be suitable forexecution on a parallel processor or a serial processor such that themethod steps may be carried out in any suitable order, orsimultaneously.

This acknowledges that software can be a valuable, separately tradablecommodity. It is intended to encompass software, which runs on orcontrols “dumb” or standard hardware, to carry out the desiredfunctions. It is also intended to encompass software which “describes”or defines the configuration of hardware, such as HDL (hardwaredescription language) software, as is used for designing silicon chips,or for configuring universal programmable chips, to carry out desiredfunctions.

Those skilled in the art will realize that storage devices utilized tostore program instructions can be distributed across a network. Forexample, a remote computer may store an example of the process describedas software. A local or terminal computer may access the remote computerand download a part or all of the software to run the program.Alternatively, the local computer may download pieces of the software asneeded, or execute some software instructions at the local terminal andsome at the remote computer (or computer network). Those skilled in theart will also realize that by utilizing conventional techniques known tothose skilled in the art that all, or a portion of the softwareinstructions may be carried out by a dedicated circuit, such as a DSP,programmable logic array, or the like.

Any range or device value given herein may be extended or alteredwithout losing the effect sought, as will be apparent to the skilledperson.

It will be understood that the benefits and advantages described abovemay relate to one embodiment or may relate to several embodiments. Theembodiments are not limited to those that solve any or all of the statedproblems or those that have any or all of the stated benefits andadvantages. It will further be understood that reference to ‘an’ itemrefers to one or more of those items.

The steps of the methods described herein may be carried out in anysuitable order, or simultaneously where appropriate. Additionally,individual blocks may be deleted from any of the methods withoutdeparting from the spirit and scope of the subject matter describedherein. Aspects of any of the examples described above may be combinedwith aspects of any of the other examples described to form furtherexamples without losing the effect sought.

The term ‘comprising’ is used herein to mean including the method blocksor elements identified, but that such blocks or elements do not comprisean exclusive list and a method or apparatus may contain additionalblocks or elements.

It will be understood that the above description of a preferredembodiment is given by way of example only and that variousmodifications may be made by those skilled in the art. The abovespecification, examples and data provide a complete description of thestructure and use of exemplary embodiments of the invention. Althoughvarious embodiments of the invention have been described above with acertain degree of particularity, or with reference to one or moreindividual embodiments, those skilled in the art could make numerousalterations to the disclosed embodiments without departing from thespirit or scope of this invention.

The invention claimed is:
 1. A computer-implemented method for assistinginput, comprising: receiving, at a processor, a plurality of key-pressesfrom a keyboard comprising alphanumeric data for input to applicationsoftware executed at the processor, a plurality of charactersrepresented on the keyboard remaining the same during each of theplurality of key presses; analyzing the plurality of key-presses at theprocessor to detect an entry in a list of common typing errors includingthe plurality of key presses; and responsive to detecting the entry in alist of common typing errors, controlling a display device to display arepresentation of at least a portion of the keyboard in association witha user interface of the application software, the representation of atleast a portion of the keyboard being displayed in proximity to a partof the display device where the entry in a list of common typing errorsis displayed separate from the keyboard, the proximity of therepresentation of at least the portion of the keyboard to the part ofthe display device where the entry in a list of common typing errors isdisplayed being at least partly dependent on the entry in a list ofcommon typing errors including the plurality of key presses.
 2. A methodaccording to claim 1, wherein the step of analyzing the plurality ofkey-presses to detect the entry in a list of common typing errorscomprises detecting whether the plurality of key-presses comprises aplurality of key-presses of a predefined key within a time threshold. 3.A method according to claim 1, wherein the step of analyzing theplurality of key-presses to detect the entry in a list of common typingerrors comprises detecting whether the plurality of key-pressescomprises a key-press of a first predefined key immediately followingone or more key-presses of further predefined keys.
 4. A methodaccording to claim 1, wherein the step of analyzing the plurality ofkey-presses to detect the entry in a list of common typing errorscomprises determining from the plurality of key-presses a key-pressinput rate and detecting that the key-press input rate is less than athreshold value.
 5. A method according to claim 1, wherein the step ofanalyzing the plurality of key-presses to detect the entry in a list ofcommon typing errors comprises accessing a stored dictionary anddetermining whether the plurality of key-presses comprise one or morewords that are not present in the stored dictionary.
 6. A methodaccording to claim 1, wherein the step of analyzing the plurality ofkey-presses to detect the entry in a list of common typing errorscomprises analyzing time delays between the key-presses and detectingwhether a user of the keyboard is not touch-typing correctly.
 7. Amethod according to claim 1, wherein the representation is partiallytransparent and is overlaid on the user interface of the applicationsoftware.
 8. A method according to claim 7, further comprising the stepof increasing transparency of the representation in dependence on a timeperiod since the entry in a list of common typing errors was detected.9. A method according to claim 1, further comprising the step ofreceiving, at the processor, sensor data from at least one sensorarranged to monitor at least a subset of keys on the keyboard to detectone or more digits of a user within a predefined distance thereof.
 10. Amethod according to claim 9, further comprising the step of: analyzingthe sensor data to determine a position of the digits of the user anddetect whether the user is not touch-typing; and responsive to detectingthat the user is not touch-typing, controlling the display device todisplay the representation of at least a portion of the keyboard inassociation with the user interface of the application software.
 11. Amethod according to claim 9, further comprising the steps of: analyzingthe sensor data to determine a position of the digits of the user;generating a model of the user's hand from the position of the digits ofthe user; and displaying the model in combination with therepresentation on the display device.
 12. A method according to claim11, further comprising the steps of: generating a further model of ahand indicating a preferred typing position, and displaying the furthermodel in combination with the model of the user's hand and therepresentation on the display device.
 13. A method according to claim 1,wherein the keyboard is one of: a hardware keyboard comprising aplurality of hardware key switches; or a software keyboard displayed onthe display device, and wherein the display device comprises a touchsensitive screen.
 14. A computer device, comprising: a processor; adisplay device; a keyboard comprising a plurality of keys and arrangedto provide key activation data to the processor; and at least one sensorarranged to monitor at least a subset of the keys to detect an objectwithin a predefined distance thereof, and provide data to the processorindicating detection of the object within a predefined distance of afirst key prior to activation of the first key, wherein the processor isarranged to control the display device to display a representation of aportion of the keyboard comprising the first key and at least one otherkey responsive to receiving the detection data, the at least one otherkey comprising the representation of a portion of the keyboard beingbased at least in part on a frequency of detected typing errors.
 15. Adevice according to claim 14, wherein the representation of the portionof the keyboard comprises a highlighted representation of the first key.16. A device according to claim 14, wherein the representation of theportion of the keyboard is centered on the first key and comprisessurrounding keys within a predefined radius of the first key.
 17. Adevice according to claim 14, wherein the predefined distance is suchthat the object is detected only when it is touching the first key. 18.A device according to claim 14, wherein the at least one sensorcomprises at least one of: a capacitive sensor mounted on each of thekeys; an infra-red detector mounted on each of the keys; and a laserarranged to emit a laser sheet over the keys and a camera arranged todetect reflections caused by the object interrupting the laser sheet.19. A device according to claim 14, wherein the keyboard is a physicalkeyboard comprising a plurality of capacitive sensors mounted to thekeyboard below a plurality of key caps.
 20. A computer device,comprising: a processor; a display device; a keyboard comprising aplurality of keys and arranged to provide key-press data to theprocessor, wherein each of the keys comprises a sensor arranged todetect a user's digit within a predefined distance of a respective keyand provide data to the processor indicating detection of the user'sdigit prior to activation of the respective key, a plurality ofcharacters represented respectively on the plurality of keys remainingthe same during provision of the key-press data; and wherein theprocessor is arranged to receive a plurality of key-presses from thekeyboard comprising alphanumeric data for input to application softwareexecuted at the processor, analyze the plurality of key-presses, detectat least one typing error including the plurality of key presses, and,responsive thereto, control the display device to display arepresentation of at least a portion of the keyboard overlaid on a userinterface of the application software on the display device, atransparency of the representation being based at least in part on afrequency of the detected typing errors.